Encrypted external HDD in ExFat?

If you were able to encrypt it using OS X built-in software, Windows would not be able to decrypt it. If you use Windows build-in encryption software, OS X would not be able to decrypt it.

You need to find a 3rd party package that works on BOTH OS X and Windows that can encrypt and decrypt on both platforms.

In the past I would have pointed to TruCrypt, however, it has been abandoned by original unknown authors, but there have been a few replacements based on the original TruCrypt source. Try Googling "TruCrypt Alternatives" and you might find something where they support both OS X and Windows.

Otherwise you will need to find another cross platform encryption package.

<bump> I would suggest looking into true hardware encryption (e.g. Apricorn Aegis Secure Key). I would reformat the device or hard drive (if not formatted already) to exFAT, which will handle large files natively on the Mac and PC for read/write. The encryption is entirely managed by the device...that is you will not need to run any other encryption on the host computer. It would be redundant and of little value to add additional encryption beyond the device's offering. TruCrypt and other software encryption methods are prone to a number of issues in terms of portability and access on other machines. This will prevent the need to install additional software on a system in order to access your files.